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Abstract 

Background: Practice guidelines (PGs) are systematically developed statements intended to assist in patient and 
practitioner decisions. The AGREE II is the revised tool for PG development, reporting, and evaluation, comprised of 
23 items, two global rating scores, and a new User's Manual. In this study, we sought to develop, execute, and 
evaluate the impact of two internet interventions designed to accelerate the capacity of stakeholders to use the 
AGREE II. 

Methods: Participants were randomized to one of three training conditions. Tutorial-participants proceeded 
through the online tutorial with a virtual coach and reviewed a PDF copy of the AGREE II. Tutorial + Practice 
Exercise'-in addition to the Tutorial, participants also appraised a 'practice' PG. For the practice PG appraisal, 
participants received feedback on how their scores compared to expert norms and formative feedback if scores fell 
outside the predefined range. AGREE II User's Manual PDF (control condition)'-participants reviewed a PDF copy of 
the AGREE II only. All participants evaluated a test PG using the AGREE II. Outcomes of interest were learners' 
performance, satisfaction, self-efficacy, mental effort, time-on-task, and perceptions of AGREE II. 

Results: No differences emerged between training conditions on any of the outcome measures. 

Conclusions: We believe these results can be explained by better than anticipated performance of the AGREE II 
PDF materials (control condition) or the participants' level of health methodology and PG experience rather than 
the failure of the online training interventions. Some data suggest the online tools may be useful for trainees new 
to this field; however, this requires further study. 



Background 

Evidence-based practice guidelines (PGs) are systemati- 
cally developed statements aimed at assisting clinicians 
and patients to make decisions about appropriate 
healthcare for specific clinical circumstances [1] and to 
inform decisions made by policy makers [2-4]. While 
PGs have been shown to have a moderate impact on 
behavior [5], their potential for benefit is only as good 
as the PGs themselves [6-8]. The AGREE II, a revised 
version of the original tool [9], is an instrument 
designed to direct the development, reporting, and eva- 
luation of PGs [10-13]. The AGREE II consists of 23 
items grouped into six quality domains, two overall 
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assessment items, and extensive supporting documenta- 
tion to facilitate its appropriate application (i.e., User's 
Manual). 

International adoption of the original AGREE Instru- 
ment and interest in the revised version has been signifi- 
cant, and attests to the potential value of this tool [14]. 
The AGREE II was designed for many different types of 
users and for users with varied expertise. Given the 
breadth and heterogeneity of the AGREE II's stakeholder 
group, efforts to promote and facilitate its application 
are complex. The internet is a key medium to reach a 
vast, varied, and global audience. However, passive inter- 
net dissemination alone, even with a primed and inter- 
ested audience, will not fully optimize its application 
and use. Our interest was to explore educational inter- 
ventions and to leverage technical platforms to acceler- 
ate an effective application process. 
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E-learning (internet-based training) provides a poten- 
tially effective, standardized, and cost-efficient model for 
training in the use of AGREE II. A recent meta-analysis 
and systematic review showed large effect sizes for inter- 
net-based instruction (clinical and methodological con- 
tent areas) on outcomes with health-profession learners 
[15,16]. Improved learning outcomes seemed to be asso- 
ciated with designs that included interactivity, practice 
exercises, repetition, and feedback. Thus, e-learning 
appeared to be a promising solution for our context. 
While the evidence base underpinning the efficacy and 
design principles of e-learning training materials are 
well established [17-23], there remain questions regard- 
ing the optimal application and combination of these 
principles for particular interventions. In this study, we 
wanted to design and test two e-learning interventions, 
a tutorial alone versus a tutorial plus an interactive 
practice exercise, against a more traditional learning 
form to determine their impact on outcomes related to 
the AGREE II. 

Our primary research question is, whether compared 
to just reading the User's Manual, does the addition of 
an online tutorial program, with or without a practice 
exercise with feedback, improve learners' performance 
and increase learners' satisfaction and self-efficacy with 
the AGREE II? Based on the results of systematic 
reviews [15,16], we hypothesized the training platform 
that included the tutorial plus the practice exercise with 
feedback would be superior to the User's Manual alone. 
For exploratory purposes, we also examined whether dif- 
ferences existed across the outcome measures between 
the two e-learning intervention groups. 

Methods 

This study was funded by the Canadian Institutes of 
Health Research and received ethics approval from the 
Hamilton Health Sciences/Faculty of Health Sciences 
Research Ethics Board (REB #09-398; McMaster Univer- 
sity, Hamilton, Ontario, Canada). Key evidence-based 
principles in the science of technical training, multime- 
dia learning, and cognitive psychology were used to 
develop the two training platforms [17-23]. 

Study design and intervention 

A single factorial design with three levels of training 
intervention was implemented (see Figure 1). 

Tutorial 

Participants received access to a password-protected 
website where they were presented with a seven-min- 
ute multimedia tutorial presentation with an overview 
of the AGREE II conducted by a 'virtual coach.' Follow- 
ing the tutorial, the participants were granted access 
to a PDF copy of the AGREE II and were instructed 



to review the User's Manual before proceeding to the 
test PG. 

Tutorial + practice exercise 

Participants received access to a password-protected 
website where they received the same tutorial presenta- 
tion described above and access to the AGREE II User's 
Manual. They were then presented with the practice 
exercise that required participants to read a sample or 
'practice' PG and appraise it using the AGREE II. Upon 
entering each AGREE II score, participants were pro- 
vided immediate feedback on how their score compared 
to the mean of four experts. If their score fell outside a 
predefined range, participants received two-stage forma- 
tive feedback to guide the appraisal process. At the con- 
clusion of their review, participants received a summary 
of their performance in appraising the practice PG com- 
pared to expert norms. Participants then proceeded to 
read and appraise the test PG. 

User's manual 

Participants assigned to the control condition received 
PDF copies of the AGREE II User's Manual for review 
before proceeding to the test PG. The User's Manual is 
a 56-page document. It provides an overview of the 
AGREE enterprise and general instructions on how to 
use the tool. Then, for each of the 23 core items, it pre- 
sents a definition of the concept and examples, advice 
on where the information can be found within a PG 
document, and the specific criteria and considerations 
for scoring. It concludes with the two global rating 
measures. 

Participants and process 

Following our sample size calculation reported in the 
detailed protocol previously published [14], we required 
20 participants per group to have at least 80% power to 
detect a performance advantage of as little as ± 0.79 
standard deviations for either of the intervention groups 
compared to the passive learning group. Methodologists, 
clinicians, policy makers, and trainees were sought from 
guideline programs, professional directories, and the 
Guidelines International Network (G-TN) community. 
Because our previous research showed virtually no dif- 
ferences in AGREE II performance as a function of type 
of users, we did not account for this factor in our study 
design [11-13]. 

A total of 107 interested individuals registered with 
the Scientific Office. After receiving a letter of invitation 
and screening for their eligibility, 87 participants were 
randomized to one of the three training conditions 
using a computer-generated randomization sequence 
(1:1:1 ratio). Individuals were eligible for study participa- 
tion if they had no or limited experience and exposure 
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Control Group 



Step 1 

AGREE II (PDF) 



Step 2 

Review/Assess 
Test-PG 




Tutorial Group 




Stepl 

Tutorial 



Tutorial + Practice Exercise Group 



Step 3 

Practice 
Exercise 



Step 4 

Review/Assess 
Test-PG 




Figure 1 Study description 



to the original AGREE Instrument or the AGREE II. To 
assess this, participants were asked to first complete an 
online eligibility questionnaire. Here, they were asked 
about the type(s) of previous experience they had with 
the original AGREE and AGREE II (as a tool to inform 
guideline development, guideline reporting, guideline 
evaluation, and other) and the extent of this experience 



(never, 1 to 5 guidelines, 6 to 10 guidelines, 11 to 15 
guidelines, 16 to 20 guidelines, 20+ guidelines). They 
were also asked if they had participated in any AGREE- 
related research study previously (yes, no, uncertain). 
Participants who answered they had not participated in 
an AGREE-related research study and who had little to 
no AGREE or AGREE II experience (defined as never 
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using either instrument or using it on a maximum of 1 
to 5 guidelines) were eligible to participate. 

These individuals were then randomized to group and 
received access to an individualized password-protected 
web-based study platform. Participants completed their 
specific training intervention, evaluated one of ten test 
PGs using the AGREE II, and completed a series of 
post-test Learner's Scales and a demographics survey. 
Participants were blinded to the study conditions, our 
research questions, and hypothesis. 

Materials and instruments 

Practice guidelines 

Eleven PGs were selected for this study: one served as 
the practice PG for participants randomized to the 
Tutorial + Practice Exercise group and, to facilitate gen- 
eralizability of results, the remaining ten were selected 
for the test PGs. Participants were randomized to one of 
the ten test PGs. Practice guideline was not a factor of 
analytic interest. Eligibility criteria for the 11 PGs are 
described in detail in the previously published protocol 
and include: English-language documents published 
from 2002 onward; were within the clinical areas of can- 
cer, cardiovascular, or critical care; were 50 pages or 
less; and represented a range of quality [14]. 

AGREE II performance 

The AGREE II consists of survey items and a User's 
Manual [11-13]: twenty- three items are grouped into six 
domains of PG quality: scope and purpose, stakeholder 
involvement, rigour of development, clarity of presenta- 
tion, applicability, and editorial independence. Items are 
answered using a 7-point response scale ('strongly dis- 
agree' to 'strongly agree'). Standardized domain scores 
are calculated enabling construction of a performance 
score profile permitting direct comparisons across the 
domains or items. The AGREE II survey items conclude 
with two global measures answered using a 7-point 
scale: one item targeting the PG's overall quality and the 
second targeting the appraiser's intention to use the PG. 
The User's Manual provides explicit direction for each 
of the 23 and two overall items, as noted above. Partici- 
pant performance served as the primary outcome. 

Learner's scale 

In addition to the primary outcome of performance on 
the test PG, a series of secondary measures, known as 
the Learner's scale, were also collected. This scale was 
comprised of Learner Satisfaction scale (i.e., satisfaction 
with learning opportunity), Self-Efficacy scale (i.e., belief 
one can succeed), Mental Effort scale (i.e., cognitive 
effort to complete a task), and Time-on-Task. With the 
exception of Time-on-Task, which was a self-report 



measure, a 7-point response scale was used to answer 
the remaining items. The questions included in the 
Learner's scale were inspired by previous work done in 
this field [17-23]. Specific reliability and validity testing 
of the items and subscales was not undertaken. 

AGREE II perceptions 

Participants were asked to rate the usefulness of the 
AGREE II (for development, reporting, and evaluation) 
and the User's Manual using a 7-point scale. 

Demographics and AGREE II Experience scale 

Participants were asked about their backgrounds includ- 
ing experience with the PG enterprise, the original 
AGREE instrument and the AGREE II. 

Outcomes and analyses 
Primary measures 

Two performance measures served as the primary out- 
comes. First, the Performance - Distance Function cal- 
culates the difference between the domain scores of the 
participants from those of expert norms. Expert norms 
were derived by members of the AGREE Next Steps 
research team who appraised the test PGs used in this 
study. Four expert appraisers rated each guideline. Mean 
standardized scores were used to construct the expert 
performance score profiles. Thus, the measure of dis- 
tance (i.e., difference in scores between participants and 
experts) for each AGREE II domain was calculated by 
squaring the difference between the participants' profile 
domain ratings from the experts' profile domain ratings. 
A series of one-way analysis of variance tests were sub- 
sequently calculated to examine differences in distance 
function as a function of training intervention. 

Second, performance was measured by examining the 
proportion of participants who met minimum perfor- 
mance competencies with the AGREE II tool [14]. A 
Pass/Fail algorithm designed for another study [14] was 
used here to calculate the performance level for partici- 
pants randomized to the condition with the practice PG. 
Secondary measures 

The Learner's scale served as the core secondary measure. 
To this end, a series of multivariate one-way analysis of 
variance tests were conducted to examine differences in 
participants' satisfaction, self-efficacy, and mental effort as 
a function of training intervention. A series of analysis of 
variance tests were conducted to examine differences in 
participants' self-reported Time-on-Task and in partici- 
pants' reported perceptions of the AGREE II. 

Results 

There were no changes to any of the outcomes once the 
trial commenced. 
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Participants (Table 1 and Figure 2) 

Letters of invitation were sent to 107 participants, of 
which 87 were eligible to participate (12 were excluded 
based on past experience with the AGREE Instrument and 
eight were non-respondents to the letter of invitation). 
Sixty participants completed the study (response rate = 
69%), 20 per condition. The majority of participants were 



female, between the ages of 25 and 65, and with some 
level of health methods training. 

Performance - distance function (Table 2) 

There were no significant differences in any of the 
domain distance functions between the three training 
groups (p > 0.05 for all comparisons). 



Table 1 Demographics 







Group 1: 
Tutorial 


Group 2: 
Practice Exercise 


Group 3: 
Control 


Gender 




% Female 


75% 


60% 


55% 


Age 




1 8 to 24 


0% 


20% 


0% 




25 to 34 


35% 


15% 


15% 




35 to 44 


15% 


30% 


35% 




45 to 54 


30% 


30% 


30% 




55 to 64 


20% 


15% 


20% 


Participants' Training 


Education 




Bachelors 


95% 


65% 


80% 




Masters 


45% 


50% 


45% 




PhD 


25% 


15% 


5% 




Physician 


30% 


35% 


30% 


Registered Nurse 


15% 


20% 


15% 


Allied Health (e.g., 


PT, OT, RT) 


5% 


10% 


0% 


Other (non specified) 


0% 


10% 


5% 


% with health research methods training 


85% 85% 100% 


Previous Experience 


Use of AGREE as a tool to inform PG development 




Never 


78% 


65% 


65% 


1 


to 5 times 


10% 


25% 


22% 


Use of AGREE as a tool to inform PG reporting 




Never 


75% 


74% 


76% 


1 


to 5 times 


15% 


10% 


17% 


Use of AGREE as a tool to evaluate PG 




Never 


71% 


61% 


48% 


1 


to 5 times 


16% 


26% 


35% 


Use of AGREE II as a tool to inform PG development 




Never 


94% 


94% 


84% 


1 


to 5 times 


6% 


6% 


10% 


Use of AGREE II as a tool to inform PG reporting 




Never 


97% 


1 00% 


84% 


1 


to 5 times 


3% 


0% 


10% 


Use of AGREE II as a tool to evaluate PG 




Never 


94% 


94% 


84% 


1 


to 5 times 


6% 


6% 


10% 
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Enrollment 



Assessed for eligibility (n=1 07) 



Excluded (n=20) 

♦ Not meeting inclusion criteria (n=12) 

♦ Declined to participate (n=0) 

♦ Other reasons (n=8) 



Randomized (n= 87) 



y 

Allocated to "overview tutorial" 
Intervention (n= 29) 

♦ Received allocated intervention 

(n=29) 

♦ Did not receive allocated 

intervention (n=0) 



t 

Allocation 



Allocated to "practice exercise" 
intervention (n= 30) 

♦ Received allocated intervention 

(n= 30) 

♦ Did not receive allocated 

intervention (n=0) 



T 

Allocated to Control Group (n= 28) 

♦ Received allocated intervention 

(n= 28) 

♦ Did not receive allocated 

intervention (n=0) 



Lost to follow-up (n=5) 

Discontinued intervention (n=4; no 
time to complete study) 



Analysed (n=20) 

♦ Excluded from analysis (give 

reasons) (n=0) 



Figure 2 CONSORT flow diagram 



Follow-Up 



Lost to follow-up (n=7) 

Discontinued intervention (n=3; 
technical difficulties, too long, no 
time to complete study) 



Analysis 



Analysed (n=20) 

♦ Excluded from analysis i 

reasons) (n=0) 



Lost to follow-up (n=4) 

Discontinued intervention (n=4; no 
time to complete study, technical 
difficulties) 



Analysed (n=20) 

♦ Excluded from analysis i 

reasons) (n=0) 



Performance - pass/fail criteria 

86% of the individuals in the Tutorial + Practice Exer- 
cise training intervention arm passed the online training 
with the practice PG. 

Training satisfaction and self-efficacy (Table 3) 

Participants reported high levels of training satisfaction 
(means 6.0+) and self-efficacy (means 5.4+). There were 
no significant differences in any measure as a function 
of training condition (p > 0.05 for all comparisons). The 
Tutorial, Tutorial + Practice Exercise, and review of the 
PDF training options were recommended by 80%, 60%, 
and 60% of participants, respectively (p > 0.05 for all 
comparisons). 



Mental effort (Table 4) 

The multivariate analysis of variance failed to show a 
difference in participants' reporting of mental effort as a 
function of training condition. With the exception of 
one measure (the AGREE II was mentally demanding), 
the univariate analyses of variance also failed to show 
significance differences. 

Time-on-task (Table 5) 

There were no significant differences as a function of 
training condition in the time spent by participants 
reviewing either the PDF version of the AGREE II or in 
the time taken to complete the test PG (p > 0.05 for all 
comparisons). 
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Table 2 Distance function (mean (standard deviation))* 



Domain Overview Tutorial Practice Exercise 


Control 




Sig 


Domain 1. Scope and Purpose 3.21 (2.96) 


2.61 (2.96) 


1 .90 (2.64) 




0.36 


Domain 2. Stakeholder Involvement 1.68(2.63) 


2.03 (2.36) 


1.71 (2.54) 




0.89 


Domain 3. Rigour of Development 1.90 (3.26) 


1 .85 (3.20) 


1 .02 (1 .40) 




0.53 


Domain 4. Clarity of Presentation 0.93 (1.1 1) 


2.86 (4.43) 


2.14 (2.15) 




0.12 


Domain 5. Applicability 3.03 (3.77) 


1 .92 (2.83) 


2.05 (2.28) 




0.45 


Domain 6. Editorial Independence 3.17 (4.41) 


2.84 (4.66) 


1 .86 (3.50) 




0.60 


*Distance Function = (mean domain score experts - mean domain score of participants) 2 Larger numbers denote greater difference between participant and 
expert performance. 


AGREE II perceptions (Table 6) Discussion 

Participants reported favourable perceptions about the l n this study, we tested two internet-based electronic 
AGREE II as a tool to facilitate the development, report- training interventions against a traditional training 
ing, and evaluation of PGs; they also reported favourable method using a PDF version of the User's Manual to 
perceptions about the AGREE II User's Manual in determine their effects on various measures related to 
enhancing skills with its application. No significant dif- performance on and attitudes toward the AGREE II. 
ferences were found for any outcome as a function of The goal was to identify the best strategy to facilitate 
training intervention conditioSn. the AGREE It's appropriate and effective uptake by its 


Table 3 Training Satisfaction and Self-Efficacy Ratings (1 to 7 scale; means and (standard deviations)). 




Training Satisfaction and Self-Efficacy 


Overview 
Tutorial 


Practice Control 
Exercise 


Univariate 
Sig 


Training Satisfaction (MANOVA, p > 0.05) 


The training exercise was conveyed at the appropriate level 


5.85 (1.09) 


6.16 (0.77) 


5.95 
(0.83) 


0.67 


The training exercise was a valuable learning experience 


6.10 (0.78) 


6.35 (0.75) 


5.75 
(1.12) 


0.15 


The training exercise was a positive experience 


6.00 (0.80) 


6.05 (0.95) 


5.45 
(1.32) 


0.09 


The training exercise was completed in a reasonable amount of time 


5.10 (1.74) 


4.60 (1 .96) 


5.65 
(1.23) 


0.16 


The training exercise has increased my understanding of the content of the AGREE II 


6.45 (0.83) 


6.40 (1 .00) 


6.25 
(0.79) 


0.77 


The training exercise has increased my confidence to assess the quality of PGs using 

the AGREE II 


5.85 (0.99) 


5.95 (0.95) 


6.00 
(0.80) 


0.87 


1 was able to navigate the training exercise with ease 


6.30 (1.03) 


6.11 (0.94) 


6.15 

(1.09) 


0.89 


The information in the training exercise was logically grouped together 


6.35 (0.81) 


6.45 (0.76) 


6.30 
(0.80) 


0.85 


The training exercise achieved its stated objectives. 


6.25 (0.72) 


5.95 (0.95) 


5.80 
(0.95) 


0.26 


The training exercise was relevant to my practice/goals and my learning needs. 


6.10 (0.85) 


6.30 (0.98) 


5.70 
(1.13) 


0.15 


Overall, 1 was satisfied with my AGREE II training experience 


6.15 (0.75) 


6.05(1.15) 6. 


1 (0.72) 


0.98 


Self-Efficacy (MANOVA, p > 0.05) 


1 am confident in my ability to use the AGREE II to assess PGs 


5.20 (1.28) 


5.25 (0.91) 


5.60 
(1.10) 


0.47 



am comfortable with the structure of the AGREE II 5.80(0.77) 6.10 (0.72) 6.05 0.38 

(0.69) 



I am comfortable with the content of the AGREE II 5.80 (0.83) 5.65 (0.59) 6.00 0.31 

(0.73) 



I am confident in applying my AGREE II skills 



5.20 (1.20) 5.20 (1.05) 5.65 0.29 

(0.99) 
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Table 4 Mental Effort Ratings (1 to 7 scale; means (standard deviations)). 



Mental Effort (MANOVA, p > 0.05) 


Tutorial 


Practice 
Exercise 


Control 


Univariate 
Sig 


Mental effort tutorial: The AGREE II Overview Tutorial was mentally demanding 


3.65 
(1.50) 


2.85 (1.81) 




0.14 


Mental effort tutorial: The pace of the AGREE II Overview Tutorial was hurried/rushed 


2.95 

(,1.3/J 


2.60 (1.61) 


- 


0.54 


At the end of the AGREE II Overview Tutorial, 1 was discouraged 


2.30 
(1 -46) 


2.15 (1.53) 


- 


0.75 


Reviewing the AGREE II was mentally demanding 


4.40 
[ I -5U) 


3.10 (1.25) 


4.37 

( I .40) 


0.006 


At the end of reviewing the AGREE II, 1 was discouraged 


3.20 
(1 77) 


2.30 (1.53) 


2.25 


0.10 


The interactive practice exercise was mentally demanding 




4.55 (1.40) 






At the end of the interactive practice exercise, 1 was discouraged 




3.16 (1.80) 






Rating and assessing the practice guideline with the AGREE II was mentally demanding 


4.85 
(1.50) 


4.70 (1 .46) 


5.05 
(1.54) 


0.76 


Rating and assessing the practice guideline with the AGREE II was very hard work 


4.25 
(1.69) 


3.90 (1 .56) 


3.50 
(1.91) 


0.39 


At the end of rating and assessing the practice guideline with the AGREE II, 1 was 
discouraged 


2.95 
(1.57) 


2.65 (1.63) 


2.30 
(1.17) 


0.38 



stakeholders. In contrast to our hypotheses, participants 
randomized to the training condition that included the 
Tutorial + Practice Exercise did not demonstrate super- 
ior performance with the AGREE II, greater satisfaction 
with the training experience, higher levels of self-effi- 
cacy, or more positive attitudes toward the tool than did 
participants randomized to the other two conditions. 

One potential explanation is that our randomization 
did not work properly, and there were differences in 
experience participants had in health research metho- 
dology and/or the AGREE or the AGREE II. Our demo- 
graphic data (see Table 2) suggest participants allocated 
to the control condition may have been more apt to 
have had minimal exposure than no exposure to the 
tools than were participants allocated to the other con- 
ditions. The inclusion of direct pretest measures to 
more accurately capture guideline performance before 
training exposure and to ensure baseline characteristics 
of the participants do not vary on this factor may be 
warranted in future studies. 

A second potential explanation for our findings is that 
our interventions did not work. This explanation, how- 
ever, is not well supported. First, each intervention arm 
aligned with design characteristics found in other stu- 
dies and systematic reviews to be effective training 



features, such as immediate feedback, interactivity, and 
repetition [15,16]. Second, albeit the data are subjective, 
they do show that participants liked all of our interven- 
tions; for example, satisfaction measures and self-efficacy 
measures are extremely high, well above the mid-point 
of the 7-point response scale. To that end, one may 
conclude then, that our control condition (i.e., review of 
the PDF version of the AGREE II only) was very effec- 
tive, and that there is a ceiling effect on performance 
measures and other outcomes. 

Exploring these conclusions further, a significant com- 
ponent in the revision of the AGREE II was the rework- 
ing of the User's Manual and its written training 
resource component. As described, the document pro- 
vides descriptions, examples, and explicit direction for 
how to evaluate a PG report using AGREE II. The com- 
prehensive nature of the PDF version of the AGREE II 
User's Manual may be quite sufficient for many poten- 
tial users. In fact, previous research, as was found in this 
study, demonstrates high support for the User's Manual 
by participants [13]. 

While this study failed to demonstrate superiority of 
the online electronic training interventions, we do not 
believe they should be abandoned all together. While we 
were successful in screening participants so that they 



Table 5 Time-on-Task (minutes; means (standard deviations)). 



Time-on-Task 


Overview Tutorial 


Practice Exercise 


Control 


Sig 


User's rating of how long it took to overview PDF copy of AGREE II 


31.70 (24.97) 


29.1 (22.83) 


38.4 (18.30) 


040 


User's rating of how long it took to do interactive practice exercise 




51.9 (30.05) 






User's rating of how long it took to read and rate PG 


70.50 (52.91) 


61 .9 (29.48) 


75.55 (50.02) 


0.63 
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Table 6 AGREE II Perceptions (1 to 7 scale; means and (standard deviations)). 



AGREE II Perception 


Overview Tutorial 


Practice Exercise 


Control 


Sig 


believe the AGREE II will be a useful tool to inform practice guideline development 


6.45 (0.605) 


6.20 (0.70) 


6.25 (0.72) 


0.47 


1 believe the AGREE II will be a useful tool to inform practice guideline reporting 


6.40 (0.60) 


6.25 (0.79) 


6.05 (0.76) 


0.31 


believe the AGREE II will be a useful tool to evaluate practice guidelines 


6.50 (0.61) 


6.50 (0.61) 


6.25 (0.72) 


0.37 


1 believe the User's Manual enhanced my skill in use of applying the AGREE II 


5.90 (0.80) 


5.95 (1.05) 


6.00 (0.92) 


0.94 



had little-to-no experience with the AGREE II or the 
original version of the tool, virtually all participants had 
some experience in health methods (e.g. systematic 
review, critical appraisal) and many had experience with 
the PG enterprise (see Table 1). This selection bias may 
represent a limitation to the study that also compro- 
mises the interpretability of the findings. Specifically, it 
may be that the online training interventions would be 
of benefit to the truly novice participant: individuals 
with no experience with the AGREE II, PGs in general, 
or health research methodology-for example, trainees 
and students in the field of health services research. 
There are some previous data to support this. In the 
separate project that developed the pass-fail algorithm 
used in this study, most of the participants were trainees 
early on in their post-graduate career with considerably 
less experience in health methods or PGs. In contrast to 
pass rates of 86% reported in this study, the initial pass 
rates for those participants was 73%, suggesting the 
training may be better suited for novice users. Future 
research studies recruiting these types of participants 
are warranted. 

Indeed, educational research supports the notion of 
adapting instructional methods based on individual dif- 
ferences in prior knowledge. In general, the literature 
suggests that good instructional design techniques may 
be of more importance for low prior knowledge than for 
high prior knowledge learners [19,22]. Redundant con- 
tent should usually be eliminated for more experienced 
learners. It is possible that the more knowledgeable lear- 
ners in our study experienced unnecessary extra cogni- 
tive load from the additional e-learning instructional 
interventions, when the control materials of the User's 
Manual were sufficient. There may even be expertise 
reversal effects, where a given instructional method that 
works well for novice learners [24] is less effective or 
even detrimental for individuals with more expertise 
[25]. In this study, it is possible that either the ceiling 
effect or detrimental effects of redundancy may have led 
to no difference from the control condition. Further 
investigation is required to assess whether efficient 
instruction on the AGREE II for more advanced learners 
will require different methods than training designed for 
entry-level learners. 

In summary, our study did not demonstrate our two 
online AGREE II electronic training interventions 



improved outcomes over the control condition. We 
believe this can be explained in part by the better than 
expected performance of the control condition (i.e. cur- 
rent standard of the PDF AGREE II, namely the User's 
Manual) and in part by the level of experience among 
the participants with health methods and PGs. Future 
research may demonstrate that the two online training 
interventions may be best suited to and effective tools 
for very novice users, new to the area of PGs and the 
AGREE II Enterprise. The training interventions are 
available through the AGREE Enterprise Web site [26]. 
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